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INTRODUCTION 



Cognitive skills and educational credentials are strong predictors of success in the labor 
market (Kerckhoff, Raudenbush, & Glennie, 2001). Unfortunately, a large proportion of 
the U. S. workforce does not have the skills and credentials required to secure and hold 
jobs that pay good wages and provide benefits (Comings, Reder, & Sum, 2001). Since 
most workers are outside the K-12 system, investments in improving primary and 
secondary schools have little immediate impact on the skills and credentials of the 
workforce. An investment in K-12 education only leads to employment outcomes after 
10 to 12 years, or even longer, while the impact for adult education could be immediate. 

To address this need, the Workforce Investment Act’s (WIA) Title II 1 supports 
educational services for adults; these services include adult literacy, adult basic education 
(ABE), family literacy, workplace literacy, adult English for speakers of other languages 
(ESOL), adult secondary education (ASE), and General Education Development (GED) 
programs. The benefits of successful participation in these programs accrue not only to 
our economy but to other aspects of our national life as well. The skills needed to succeed 
in the workforce are the same skills needed to succeed as parents and citizens. In 
addition, workers, parents and citizens must now continue learning throughout their lives. 

A strong foundation of basic skills and knowledge supports lifelong learning. 

Adult literacy practitioners must make many decisions about how to design their 
programs to serve the goals of WIA. Practitioners usually base decisions about the design 
of program services on theories and approaches learned in training, the opinions of 
experienced practitioners, and personal experience gained through trial and error (Beder 
& Medina, 2001; Smith, Hofer, Gillespie, Solomon, & Rowe, 2003). Though these 
decisions often lead to programs that meet the accountability requirements of WIA, the 
practices employed by programs usually have not been rigorously evaluated. 

WIA requires programs to base decisions about the design of programs on evidence 
from scientific research. The Department of Education defines scientific research as studies 
that are rigorous, systematic, objective, empirical, and peer reviewed and that rely on 
multiple measurements and observations, preferably through experimental or quasi- 
experimental methods. 2 This evidence-based approach to making decisions is meant to 
insure that participants in adult literacy programs receive services that provide them with 
the best possible opportunity to gain the skills and credentials workers need to be 
successful in the labor market and in their roles as parents and citizens. 



1 http://www.doleta.gov/usworkforce/wia/wialaw.txt 

2 http://www.northwestern.edu/ipr/events/lectures/DPPL-Whitehurst.pdf 
http://www.ed.gov/rschstat/research/pubs/ies.pdf 
http://www.ed.gov/nclb/methods/whatworks/eb/evidencebased.ppt 
http://www.ed.gov/rschstat/research/pubs/rigorousevid/index.html 
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Unfortunately, the field of adult literacy only has a small foundation of scientific 
research. Beder (1999) reviewed 115 studies of program outcomes and impact in the field 
and found that only 23 met minimum standards for analysis. Once his review was 
complete, he concluded that even these 23 studies had flaws that significantly 
compromised the validity and usefulness of the findings. Kruidenier (2002) identified 
only 70 adult reading research studies that met the standards of scientific research. 
Comings, Soricone, and Santos (2006) reviewed the literature on advice for the design of 
services (including such factors as management, intake, orientation, support services, 
learning resources, and reengagement of dropouts, but excluding instruction of specific 
content such as reading, writing, or math) and found that little scientific research 
conducted in adult literacy programs supported the advice. All three reviews 
acknowledge that the weakness of the research base is due, in part, to challenges to 
obtaining meaningful results from scientific research that plague studies in the adult 
literacy field. 

This paper has two goals that support the building of a stronger research base for 
the adult literacy field. The first goal is to interest researchers in pursuing rigorous 
scientific research in this field. Despite the many challenges to research, this field offers 
researchers some opportunities to have a positive impact on education, increase the 
research base, and advance their careers. The second goal is to identify these challenges 
and approaches to overcoming them so that future research could design more successful 
studies. Addressing these challenges to research is a critical step toward building a larger 
and stronger foundation of evidence to support practitioner decisions. 
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STRUCTURE OF "THS MONOGRAPH 



This monograph looks first at opportunities provided by research in the adult literacy 
field and then identifies challenges to scientific research and suggests ways to address 
these challenges so that better evidence can be generated in the future. The target group 
for this monograph includes present and prospective researchers but also policymakers 
and administrators who seek a better understanding of research in this field. 

Researchers in this field have discussed the challenges to rigorous research for 
decades, usually in relation to the problems they faced in individual studies. This 
monograph looks across studies to identify the challenges and suggest ways to overcome 
them. Though this monograph provides advice on how to overcome the challenges to 
scientific research in this field, it also serves as a device for focusing a discussion that 
could lead to better advice. For both the challenges to and the opportunities of research in 
this field, the monograph identifies studies that provide examples, and readers who are 
interested in pursuing research in this field could follow up on these examples by reading 
the original reports of this research. 

The preparation of this monograph began with the drafting of a discussion paper 
that summarized insights from published sources. That paper was discussed in a two-day 
workshop with a group of advisors, all of whom have been engaged in scientific research 
in this field and have reviewed the existing literature used by practitioners as sources of 
advice. The insights from that meeting formed the basis for a rough draft of this 
monograph, which was reviewed by members of that group and two researchers who 
were not able to attend the workshop. The suggestions from that review were 
incorporated in a first draft. The first draft was peer reviewed by scholars who have been 
working on the development of evidence-based practice in adult literacy, and their 
suggestions were included in this final version. The members of the group of advisors 
and the two researchers also commented on the first draft, and some but not all of their 
comments were incorporated into this final version. Three Department of Education 
officials provided comments and suggestions at several points during the preparation of 
this monograph. A list of the people who provided input into this paper is available in the 
Appendix at the end of this monograph. 
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OPPORTUNITIES PRCMDED BY RESEARCH INH-E ADULT 

LITERACY HELD 



Research in adult literacy programs faces many of the same challenges as research in K- 
12 schools, but some aspects of the adult literacy field are more advantageous to research 
than in the K-12 field and some are less advantageous. A discussion of the positive 
aspects of undertaking studies in the adult literacy field might encourage scholars to 
tackle its challenges. The three positive aspects described here are: (1) the small number 
of studies that both address important questions and apply rigorous scientific methods, 

(2) the possibility of measuring impact on several important adult roles, and (3) the ease 
of access to adult populations. 



Small Number of Studies 

Beder (1999), Kruidenier (2002), and Comings, et al. (2006) found that the field of adult 
literacy research has only a small number of studies that meet the criteria for scientific 
research. This is a problem for the field, but it could be an opportunity for researchers, 
particularly those who are at early stages in their careers. Since the field still has a wide 
array of basic questions to answer, adult literacy research offers scholars an opportunity 
to build part of the foundation for the field by answering questions that may have a 
significant impact on policy and practice. Researchers who are at early stages in their 
careers could establish a program of research that might continue for their whole career. 
In addition, research in adult literacy has the promise of having a positive impact on the 
lives of millions of adults. 

These basic questions fall into domains that include descriptions of the population 
that needs services, approaches to instruction that work well with this population, 
approaches to the delivery of services that allow adults to participate in learning with 
sufficient intensity, duration, and engagement to make meaningful progress, and ways to 
measure the learning gains and life impacts of participation in programs. A review of the 
adult literacy literature on any of these domains quickly leads to many specific questions 
that have yet to be answered. Examples of specific questions that are of interest to 
practitioners and policymakers include how to help adults successfully transition from 
adult literacy programs into postsecondary education, how best to tailor instruction to 
adults at different literacy levels, and how to measure numeracy outcomes. 

Some of these basic questions have been answered in the K-12 literature, and that 
literature provides a basis on which to build an adult literacy study. For example, in her 
analysis of the role of vocabulary instruction in adult basic education, Curtis (2006) 
points out that, despite the fact that vocabulary has long been recognized as a key factor 
in reading comprehension and as a significant variable in minority language learners’ 
reading success in the K-12 literature, few studies have focused specifically on 
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vocabulary acquisition and instruction for adult learners. She outlines several hypotheses 
about the connection between vocabulary and reading comprehension developed through 
research on K-12 students and recommends that the relevance of these hypotheses to 
adult learners be explored through future research. Similarly, Sawyer and Joyce (2006) 
report that the area of spelling is one in which little research has focused specifically on 
adult learners, although research has been conducted on school-age learners and some 
research has been done with sub-groups of adults, including those with learning 
disabilities and hearing impairments, as well as those enrolled in ESOL classes. The 
authors point to the need for future longitudinal studies of spelling development among 
adult students and experimental studies to assess the effectiveness of instructional 
methods for adults of varied literacy levels. 

While such bodies of research provide a foundation for studies in adult literacy, so 
do existing data sets. Though many national data sets are not appropriate for adult 
literacy research, some data sets (such as those developed by the National Center for 
Educational Statistics [NCES], the Division of Adult Education and Literacy, and the 
GED Testing Service) do provide opportunities to explore questions of interest to the 
adult literacy field. In addition, many states now have effective student data systems that 
could be sources for secondary analysis. Since these data have already been collected and 
put in a form ready for analysis, studies with them are easily done without significant 
financial support. Moreover, the sample sizes available for research far exceed those that 
most scholars could collect without the investment of substantial resources. 

These data sets have been the source of scientific research in the adult literacy 
field published over the last decade. For example, Hollenbeck (1996) used data from the 
National Household Education Survey and the Current Population Survey of 1991 to 
examine the benefits and costs of workplace literacy training. Mumane, Willett and 
Boudett (1995) employed data from the National Longitudinal Survey of Youth (NLSY) 
to determine whether the earnings and employment trajectories of male dropouts are 
affected by acquisition of a GED. In a later study (1997), the same authors used the 
NLSY data set to examine how dropouts’ acquisition of a GED affects the probability of 
obtaining training, participating in postsecondary education, or joining the military 
services. Murnane, Willett, and Tyler (1999) used data from the High School and Beyond 
(HS&B) data set to explore whether the labor market benefits of a GED are different for 
dropouts who leave school with very weak cognitive skills, compared with those for 
dropouts who leave with stronger cognitive skills. HS&B data also served as the basis for 
a 2003 study by Tyler, Mumane, and Willett that examined the labor market value of the 
GED credential for female dropouts. 

Soricone (2000) used data from the National Education Longitudinal Study 
(NELS) to investigate when dropouts are most likely to obtain a GED and whether 
dropouts who leave school with more years of schooling and higher cognitive skills are 
more likely to obtain their GED earlier. GED records from Florida and Texas served as 
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the basis for Tyler, Mumane and Willett’s 2004 analysis of GED re-testing regulations 
and passing score requirements. Uvin (2001) used data from the National Household 
Education Surveys of 1995 and 1999 (NHES: 95 and NHES: 99) to gain insight into the 
nature of participation in adult ESOL classes. In this study, Uvin explored whether 
knowledge of, perceived benefits from, and access to adult ESOL courses predicted 
whether non-native English speakers participated in ESOL classes and, among immigrant 
adults, whether Black, Hispanic, Asian, and other adults of color were less likely to 
participate in ESOL courses than White adults. 

Other studies have employed unique data sets that have combined information 
from previously existing state-level data sets. In a 1998 study, Tyler, Murnane and 
Willett used a unique merged data set to estimate the impact of the GED on young 
dropouts’ earnings. The data set contained the GED test scores (from the GED Testing 
Service) and earnings (from the Social Security Administration) of a sample of dropouts 
aged 16-21 who attempted the GED in 1990. This study eliminated the potential for 
selectivity bias arising from the motivation to take the test among individuals since all 
subjects had similar motivation to pursue the GED credential (therefore dropouts’ 
earnings could not be attributed to differing levels of motivation). The study involved a 
series of “natural experiments” that arose because of different standards for passing the 
GED tests across states. Thus, the authors were able to compare the earnings of 80,000 
dropouts who had similar skill and motivation levels but differed in whether or not they 
had a GED, because of differential state passing score requirements. This allowed the 
authors to focus on the labor market signaling effect of the GED credential itself. 

In a 2004 study, Tyler and Kling explored the impact of earning a GED while in 
prison on the post-release earning of criminal justice offenders. The study was based on 
a unique data set constructed for the project by drawing on data from the Florida 
Department of Corrections, the Florida Department of Law Enforcement, and the 
Florida Education and Training Placement Information Program. The master data set 
contained basic demographic, criminal justice, and test score information, as well as 
post-release quarterly earnings (1994-2000) for males who had been incarcerated in 
Florida state prisons. 

Using a unique data set that combined data from Florida’s GED and 
Unemployment Insurance administrative records, Tyler (2004) conducted a series of 
studies on the value of the GED to dropouts who pursue the credential. Using a subset of 
the Florida data set used by Tyler, Soricone (2003) examined the relationship between 
basic skills and the earnings and employment outcomes of Spanish-speaking GED 
recipients. Uvin (2003) used administrative data sets from Massachusetts for a study of 
the employment and earnings trajectories of limited English-proficient workers 
experiencing job displacement. Data for the study were drawn from the Massachusetts 
Job Training Partnership Act management information system and the Unemployment 
Insurance earnings database of the Massachusetts Department of Revenue. 
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Since, compared to the K-12 field, little scientific research on adult literacy has 
been published, researchers have an opportunity to address fundamental questions in this 
field by building on existing K-12 research, which provides researchers with 
opportunities to propose studies that employ accepted theory and draw from tested 
methodologies. Some of these questions can be answered with existing data sets that have 
not been used for adult literacy research, providing researchers with opportunities to 
undertake studies without the need for original data collection. 



Measuring Outcomes and Impact 

Most educational research is focused directly or indirectly on instructional outcomes, 
such as higher test scores or greater high school completion rates. However, education 
also has an impact on outcomes valued by other fields. Investments in education can have 
an impact on such things as the economy, tax revenues, health indicators, and crime rates, 
for example. Funding sources interested in those types of impact may be interested in 
funding adult literacy research, and journals in these fields might be interested in 
publishing articles that explore impact from participation in adult literacy programs. 

Comings et al. (2001) identify several types of hypothesized positive impact of 
successful participation in adult literacy services, including increased income and labor 
market participation, improved school performance of participants’ children, and greater 
civic participation. These hypotheses are based on correlations between the skills and 
knowledge taught in adult literacy programs (such as reading) and indicators of these 
types of positive impact. These hypotheses assume that higher skills acquired as an adult 
have the same impact as skills acquired while in the K-12 system. A small body of 
experimental studies (Hamilton, 2002 and Tyler, 2005, for example) supports some of 
these hypotheses. These types of impact are valued by policymakers and are the subject 
of research in both education and other fields. 

These types of impact also accrue from investment in the K-12 and postsecondary 
education systems (Behrman & Stacey, 1997). However, the investment that begins when 
a child enters kindergarten and ends with graduation from high school or college does not 
begin to demonstrate impact until this long process is complete and students become 
workers, parents, and active citizens. Since adult participants are usually workers, 
parents, and citizens when they begin participation, investment in successful adult 
literacy programs could lead to impact immediately or within a few years. 

In addition, some adult literacy programs focus their curriculum on specific adult 
roles. For example, family literacy programs are meant to have a direct impact on the 
school performance of participants’ children, and workplace literacy programs are meant 
to have a direct impact on employment and income. General programs sometimes focus 
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part of their curriculum on health, the environment, or community involvement. These 
program interventions might lead to immediate impact. 

Bingman, Ebert, and Smith (1999) reported on a longitudinal study of adult 
literacy participants in Tennessee from 1991 to 1995. The study set out to determine the 
impact of participation in adult literacy programs on the lives of adult students. The study 
focused on changes in the domains of work, family, and community after the individuals 
enrolled in literacy programs. Interviews with 199 individuals conducted one year after 
enrollment revealed positive change in the areas of employment, community 
involvement, and children’s education. 

Focusing on the adult role of parent. Waterman (2006) analyzed data from an 
experimental study that explored the impact of an intervention designed to promote 
parental involvement in children’s education through adult ESOL classes. The 
intervention was implemented over a six-month period after which outcomes were 
assessed. The study found that mothers in the treatment group experienced greater gains 
in both ESOL skill levels and parental involvement scores than mothers in the control 
group. Differences between the results of the two groups were statistically significant. 
Waterman’s study represents an instance where the immediate impact of adult literacy 
education was twofold, appearing within and beyond the classroom. 

Using data from the National Household Education Survey (NHES) and the 
Current Population Survey (CPS), Hollenbeck (1996) explored the economic impact of 
workplace literacy programs. He found that, even when controlling for industry and 
occupation, participation in workplace literacy was associated with an 8% to 13% 
increase in earnings. 

Economic data is often available for multiple years, allowing forjudging both 
immediate and long-term impact. For example, in a study of incarcerated dropouts, Tyler 
and Kling (2004) found that, among racial and ethnic minorities, positive effects of the 
GED observed during the first year of post-release disappeared after two years. Among 
white dropouts, no effects of the GED were observed across the three post-release years 
included in the study. In a study of Florida GED test takers, Tyler (2002) tracked 
dropouts over three years following GED examination attempts and found that those who 
scored a standard deviation higher on the GED math exam had earnings that were 6.5% 
higher than those of dropouts with lower GED math scores. 

Several studies focus on a five-year period to observe economic impact. Tyler, 
Murnane and Willett (2000), examined the skills-earnings relationship for dropouts in 
New York and Florida five years after they had attempted the GED tests. The authors 
found large earnings returns to cognitive skills for both males and females, regardless of 
whether or not dropouts had successfully obtained the GED. Tyler et al. (1998) examined 
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the economic impact of GED acquisition across a national sample of dropouts five years 
after receipt of the credential. The authors found that GED acquisition increased the 
earnings of white dropouts; however, no effects were observed for the earnings of 
minority dropouts. 

Similarly, the National Evaluation of Welfare-to-Work Strategies (NEWWS) 
tracked participants over a five-year follow-up period (Hamilton, 2002). The study 
consisted of a randomized experiment that compared the outcomes of over 40,000 
welfare recipients (predominantly single mothers) who participated in welfare-to-work 
programs with those who did not. Among the study findings, researchers noted that all 1 1 
of the NEWWS programs “increased single parents’ employment and earnings and 
decreased their welfare receipt and payments relative to the levels found in the programs’ 
absence” (Hamilton, 2002, p. 23). Despite the fact that programs helped participants 
become more self-sufficient, their total income (from work and benefits) did not increase, 
since, as work income increased, benefits decreased. 

Adult literacy programs offer an opportunity to educational researchers to partner 
with scholars from other fields and draw from research funding resources in those fields, 
when measuring impact. For example, research in recent years has established strong 
links between literacy and health outcomes. In a review of research, Rudd, Moeykens, 
and Colton (2000) report that low literacy levels have been associated with poor health 
status (Weiss, Hart, McGee, & D’Estelle, 1992, cited in Rudd et al., 2000), higher 
reported history of heart disease and diabetes (TenHave et al., 1997, cited in Rudd et al., 
2000), reduced use of health services (Baker, Parker, Williams, Clark, & Nurss, 1997, 
cited in Rudd et al., 2000), reduced use of mammography screening leading to later stage 
diagnosis of breast cancer (Davis, et al. 1996, cited in Rudd et al., 2000), and less of the 
knowledge required for self-management of chronic diseases (Williams, Baker, Parker, & 
Nurss, 1998, cited in Rudd et al., 2000). 

While little health literacy research has been conducted specifically focused on 
students in adult literacy classes, the possibility of targeting health literacy research on 
these students offers great potential. As Rudd et al. (2000) suggest, adult literacy 
programs “provide ready access to populations with low functional health literacy, and 
both teachers and students from these programs can be engaged in the strategic 
development work (formulating and testing strategies) that must take place to address 
fully the health-related needs of this target group” (p. 184). Thus, involving students in 
adult literacy classes in research on improving strategies for working with less literate 
patients could provide important insights with implications for the health outcomes of 
adult literacy students and other adults who face similar health literacy challenges. This is 
an area of research in which public health scholars and funding agencies have already 
shown an interest. 
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Drawing support from the national interest in welfare reform, research conducted 
for the NEWWS provides useful insights into the effectiveness of different program 
approaches to serving welfare clients. The study showed that “employment-focused 
programs generally had larger effects on employment and earnings than did education- 
focused programs” (Hamilton, 2002, p. 28). However, the most effective program was one 
that employed a mixed strategy, which outperformed other programs in terms of the size 
and consistency of its earnings gains, as well as increases in the stability of employment 
and earnings growth. This program’s success suggests several key features of effective 
programs: “an employment focus, the use of both job search and short-term education or 
training, and an emphasis on holding out for a good job” (Hamilton, 2002, p. 36). 

Many of the fields that could be linked to adult literacy have much greater research 
funding than does the adult literacy field. In addition to expanding the support available for 
research, partnerships across sectors open up a wider range of publication opportunities. 
Most of these relationships have not yet been formed, and those that have, between adult 
literacy and public health and labor economics, for example, are recent. However, these 
relationships offer great promise for both sides of these possible partnerships. 



Ease of Access 

Researchers working in K-12 schools face challenges in obtaining access to research 
subjects because almost all of those subjects are under the age of 18. Parents of these 
students, or school administrators acting on their behalf, must give consent to children’s 
participation in studies. Often, decisions to participate must involve several levels of an 
educational bureaucracy. For example, to conduct a study of social class influences on 
the adjustment to middle school in a rural northeastern school district, San Antonio 
(2001) was required to obtain the consent of the district superintendent. Even after 
responding to the superintendent’s initial critiques of her research design, San Antonio 
found that her request for access to students was refused. More than one year after her 
initial request, after redesigning her study as a community-based, rather than a school- 
based one — and after school administrator and school board approval — San Antonio was 
finally granted the opportunity to work directly with middle school students in two 
district schools in order to carry out her research. Had it not been for her experience and 
community contacts within the district, she would likely have had to abandon her study. 

In contrast, adult students can give their direct consent to participate in a 
research project. In addition, most adult students are participating in programs of their 
own volition, while children are required by law or by their parents to attend school. 
Adult students can make a decision about missing class time to participate in a study, 
while that decision must be made by parents and school administrators for children. 
Adult literacy researchers who reviewed this paper expressed that most programs 
were open to researchers studying their efforts. For some studies, this ease of access 



n 




NCSALL Occasional Paper 



January 2007 



may be an advantage to undertaking research in the adult literacy programs as 
opposed to K-12 schools. 

Researchers interested in the K-12 system might find working with adult literacy 
students an easier way to explore some hypotheses. Younger students in adult literacy 
programs are close to the age of high school students. Often, these young students are in 
adult literacy programs because they were not successful in the K-12 system. They may, 
therefore, be good subjects for studies that seek to help underperforming adolescents 
achieve in school. In program year 2003-2004, 372,591 (14%) of WIA Title II students 
were aged 16-18, while 677,540 (25%) were aged 19-24. Of these students, 37,917 (3%) 
aged 16-18, and 230,522 (20%) aged 19-24, were immigrants in ESOL classes (U.S. 
Department of Education, 2006). Because of the relative ease of involving them in 
research, young immigrant adults in these programs may be good subjects for studies that 
seek to help immigrant adolescents in school. 

During the early stages of a research project, a study may be easier, quicker, or 
less expensive to undertake with young participants in adult literacy programs than with 
children in schools. Approaches to instruction could be developed and tested on a small 
scale with young adults and then tested later with high school adolescents once the pilot 
effort gave initial support to the efficacy of the approach. A research and development 
effort with young students in adult literacy programs, therefore, could identify 
interventions that would be highly likely to work with adolescents in high school. A 
study with 18- and 19-year-old adult literacy students might be easier and less expensive 
than one with 16- to 17-year olds in school, though the two groups would have to be 
similar in most other ways. 

A NCSALL-supported study that will publish its results next year is an example 
of this opportunity. This experimental study is focused on adults whose reading ability 
falls in the range of grade level equivalent 5 to 8. Some of the adults in this study are not 
much older than high school students, many of whom have similar reading scores. Over 
one third (35%) of the study subjects are under the age of 22. The findings from this adult 
literacy study could inform a similar study with high school students, providing insight 
that would strengthen both the intervention and the research methodology. 
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Once convinced that adult literacy offers opportunities for studies, a researcher must still 
overcome challenges to successful implementation of research. Two discussions of 
research in the adult literacy field have identified several challenges to rigorous research 
that may be greater than they are in K-12 schools (Beder, 1999; Greenberg, 2007). The 
group of advisors looked at this literature and drew on their own experience to identify 
several additional challenges to success in research in adult literacy programs. These 
challenges are described here and suggestions for overcoming them are provided. The 
challenges include: (1) lack of theory-based models for instruction and services, 

(2) informed consent, (3) incomplete and inconsistent data, (4) inaccurate data, (5) lack of 
valid and reliable tests, (6) multiple populations, (7) lack of program resources, 

(8) sampling, (9) shortcomings of existing data sets, and (10) lack of research resources. 



Lack of Theory-based Models for Instruction and Services 

One theme that arose out of the discussion among the group of advisors was the lack of 
theory on which to develop models for instruction and services in this field. Adult 
literacy sometimes builds its models by drawing on theory for instruction from the K-12 
system. For example, years of theory development in K-12 reading instruction led to the 
testing of the components approach to teaching reading, which proved effective as a 
model for instruction for young children (Curtis, 1980, 1997; Read, 1988; Stanovich, 
1986). Since then, additional research has been able to build on this foundation. 

Adult education research is also building on this foundation. For example, 
Greenberg, Ehri, and Perin (1997, 2002) conducted research to explore the word-reading 
skills of low-literate adults. Their studies were based, in part, on theory developed with 
children and used comparisons with children to help elucidate adult learner reading and 
spelling processes. Alamprese, Tao, and Price (2003) investigated ABE instructors’ use 
of K-12, structured reading curricula with low-level adult students and found that the use 
of these curricula was associated with learners’ development of their decoding skills. The 
National Institute for Literacy (NIFL) and the National Institute for Child Health and 
Human Development (NICHD) are funding research that tests the K-12 model for 
literacy instruction with adults. 

Research in adult literacy also draws on general adult education theory, but most 
of this theory was developed through research on adults who have the skills and 
credentials that adult literacy students lack (Merriam, 2004). Research designs based on 
theory drawn from K-12 education or adult education research might be appropriate for 
some adult students but not those with the lowest literacy skills who often have 
challenges to learning, such as learning disabilities or cognitive disabilities. For example, 
Strucker and Davidson (2003) found that the lowest-level adult literacy students had 



13 




NCSALL Occasional Paper 



January 2007 



reading components profiles that looked like those of reading disabled K-12 students. 
Outcomes for these students may be constrained by their disabilities and their previous 
difficulties in school. K-12 theory developed out of research with reading disabled K-12 
students might be appropriate for designing adult education research, but theory 
developed out of research with average students might not. 

In their study of 643 low-level adult literacy students, Alamprese et al. (2003) 
asked learners to indicate whether they currently have a learning problem or disability or 
if they had one as a child. The authors reported that 41% of subjects indicated that they 
currently have a problem, and 40% noted that they had a problem as a child. Learning 
problems were more common among native-born students than immigrants. 

The lowest-level literacy students may benefit from the approaches used with 
learning disabled children, but this approach may bring benefit only to adult students who 
study for many hours each year. Adult students usually spend only a limited number of 
hours each year in programs, and they may need different outcome measures to identify 
benefit. In fact, Reder (in press) found that some participants in adult literacy programs 
do not improve their literacy skills but do make positive changes in their literacy 
practices. Changing the outcome measure from improvement in reading skills to ability to 
accomplish specific tasks might lead to intervention designs that produce more robust 
impact. For example, Silver-Pacuilla (in press) suggests that assistive technology might 
provide immediate impact (increased ability to accomplish reading and writing tasks), 
while also slowly improving literacy skills. For example, computer programs that assist 
reading with text-to- voice recognition and dictionaries could help students begin reading 
better and provide their teacher with information on specific vocabulary problems that 
could form the focus of instruction. An intervention that helps reading disabled adults use 
assistive technology to address their daily literacy needs, while still helping them 
improve their component skills, might be more productive. 

K-12 reading theory can provide the basis for further research on adult learners. 
However, K-12 theory is being tested within the existing model for adult literacy 
programs. As do K-12 schools, the existing model for literacy programs requires adults 
to attend classes at specific times and in specific places. This model also assumes that 
adults can spend hundreds or even thousands of hours in classes over several years, as do 
school children. 

Findings from the longitudinal study of adult learners in Portland, Oregon 
(Reder, in press) suggests that adults employ episodes of program participation and self- 
study over years of engagement in learning to reach their educational goals. Many adults 
study on their own, stop studying, and then join a program only to stop again. They may 
repeat a version of this pattern over many years. This research suggests that services 
might be more effective if they supported connected episodes of program participation 
and self-study. 
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Research that tests models of instruction may prove them effective with only 
those students who are able to participate in services designed on a school model. If that 
group represents a minority of the adult student population, then research may not find 
those models to be effective or find that they lead to only modest gains. However, in a 
program designed to accommodate connected episodes of program participation and self- 
study, the same instructional approaches may prove highly effective. 

One option to address this challenge would be to divide the model for programs, for 
research purposes, into two parts: 

1. Instruction of specific content (such as reading, writing, math, English, and 
preparation for the GED test). This could take the form of curriculum and 
materials that teachers and students use to improve skills and knowledge or 
expand practices. 

2. Services to support instruction (such as recruitment, intake, orientation, 
persistence support, program management, intensity and duration of 
instruction, use of technology, and transition to postsecondary education or 
training). This could take the form of program structures and processes that 
enable students to participate in instruction with sufficient intensity, duration, 
and engagement to make meaningful progress. 

The field could then adapt K-12 models for instruction and focus adult literacy research 
on effective models for services. Research, such as the Portland longitudinal study, could 
build theory on which models could be designed and tested. Once new models for 
services are developed and proven effective, research on improving instruction of specific 
content might be more productive. 

The scholarly discourse that would have developed stronger theories has been 
constrained by a low level of research funding, a small number of academics committed 
to this field, and a limited number of journals focused on adult literacy. Given this 
situation, employing K-12 and adult education theory is a reasonable approach to 
research design. However, the nature of adult literacy students and programs makes this 
adaptation problematic. 



Informed Consent 

Informed consent of subjects in a research study has always been important, but it is now 
an essential element of every study (Raich, Plomer, & Coyne, 2001). Researchers must 
“describe the research process to potential human subjects, explain the potential benefits 
and risks. . .provide assurance that no benefits or standing will be lost by non-participants, 
and clarify that consent is voluntary and may be withdrawn by the participant at any time” 
(Ross, Sundberg, & Flint, 1999, p. 2). Informed consent “generally implies that a person 
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(1) has the capacity to consent, (2) has been informed of significant ramifications 
concerning the procedure, (3) has freely and without undue influence expressed consent, 
and (4) consent has been appropriately documented” (American Psychological 
Association, 1992, Section 4.02, ( j[a). Seeking the consent of research participants allows 
them to be involved in the research process and encourages both a sense of ownership and 
an interest in the process of the research (Green, Duncan, Barnes, & Oberklaid, 2003). 

Greenberg (2007) identifies gaining truly informed consent, as opposed to gaining 
signatures on an informed consent form, as a challenge to research in the adult literacy 
field. The subjects in adult literacy studies usually have limited literacy skills or weak 
English language skills. This makes both oral and written approaches to informed consent 
difficult and sometimes impossible. 

This is, of course, an ethical challenge to research, but it is also a practical 
challenge for two reasons. First, funding agencies and the institutional review boards 
(IRB) of research institutions are seeking greater assurance that subjects are giving truly 
informed consent (Raich et al., 2001; Green et al., 2003). The procedures set up by IRBs 
may assume English literacy skills greater than those of subjects in an adult literacy 
study. The inability to provide assurance of informed consent might cause delays in the 
approval of research projects and even jeopardize funding altogether. Second, potential 
subjects who do not understand the informed consent or find it difficult may be more 
likely to say no, lowering the percentage of subjects who have the lowest language or 
literacy skills in a study sample. This might jeopardize the generalizability of study 
findings to programs that serve the lowest- level students. 

Focusing on the issue of informed consent among students and parents in 
Australia, Green et al. (2003) offer some recommendations for adapting research 
information to increase the likelihood of informed decisions about participation. These 
include expanding the number of comparisons, analogies, or examples, as well as using 
pictures, diagrams, or videos for explanations. In addition, they suggest a focus on plain 
language (using common everyday words rather than technical ones), minimizing the use 
of unnecessary information not related to main concepts, and employing direct, 
conversational writing. As the authors note, adopting such an approach can shorten the 
time between research proposal submission and approval and contributes to a positive 
change in meeting the information requirements of parents and young subjects. 

Raich et al. (2001) suggest stronger action on the part of IRBs to encourage 
researchers to obtain truly informed consent. A process to do so should include 
“asking investigators to describe what they will say to subjects, how much time is 
provided for decision making and who specifically will obtain the consent” (p. 441). 

The authors also suggest that structured teaching, explicit categorization of 
information, and direct questioning lead to both higher comprehension and better 
recall among potential study subjects. 
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In her recent study, Greenberg (2007) read the informed consent forms aloud for 
her subjects, but she points to her own previous research (Greenberg et al., 1997) that 
shows adults with low literacy skills also have low oral receptive vocabulary levels. 
Subjects may hear what they are consenting to but may not understand it. For example, 
Greenberg’s university IRB required that informed consent forms and procedures inform 
subjects that data about them would be stored in pas sword- locked computers. She felt the 
subjects in her research would not understand the concept of password-locked computers. 

Many subjects in adult literacy research are immigrants who have low oral and 
literacy skills in English. This obviously complicates the informed consent process if 
researchers must provide oral procedures and written forms in several languages. The 
ESOL Lab School at Portland State University provides an example of a way to ensure 
that students are able to provide truly informed consent for participation in research. 

One of the unique qualities of the Lab School is its use of “comprehensive 
recordings of classroom language” (Reder, 2005, p. 5). Recordings include what both 
teachers and students are doing and saying, especially to one another. In order to gather 
this essential data, the Lab School added informed consent procedures to the school’s 
standard intake and orientation process. Prospective ESOL students in the Lab School 
were shown a video in their native language and given a tour of the Lab School 
classroom and recording facilities. Prospective students were then given the opportunity 
to sign an informed consent form in their native language, which granted researchers 
permission to use students’ recorded voices and images. If at any time students were 
uncomfortable with their participation in the research, they were permitted to transfer to 
other sites within the overall program. Researchers found that very few students chose to 
leave (Reder, 2005). 

Raich et al. (2001) support such use of audiovisual materials, which, as they point 
out, provide supplementary information, improve recall, and increase both the 
satisfaction and likelihood of study participation. However, even with a well-designed 
oral or video presentation, some indication from the subjects that they understand the 
decision they are making would ensure that the consent is truly informed (Titus & Keane, 
1996, cited in Raich et al. 2001; Williams, Rieckmann, & Trenholme, 1977, cited in 
Rudd et al., 2000). That indication of understanding could come through a description by 
the subject of what it is they are consenting to do. Since a subject and a researcher may 
speak different languages, a translator may be necessary. This could be a costly and time- 
consuming process; however, it may be the best means of ensuring that subjects fully 
understand the implications of participating in research. 

Most adult literacy research is similar to K-12 research in relation to the core 
objectives of informed consent. The point of informed consent is to provide sufficient 
information to potential subjects so that they understand what they will be participating in 
while empowering individuals to “enter into a voluntary revocable agreement to 
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participate, pending actual experience once the research begins” (Ross et al., 1999, p. 171). 
A field-wide effort to develop basic informed consent media in multiple languages could 
help researchers to secure basic consent. Researchers in other fields, public health for 
example, have to address the problems of informed consent with low-literate and non- 
English speaking adults, and the adult literacy field could draw on that experience. 
Researchers would then have to complete the process with only the specifics of their study. 



Incomplete and Inconsistent Data 

Beder (1999) and Greenberg (2007) identify incomplete and inconsistent data as a 
challenge to research in the adult literacy field. Some well-resourced studies are able to 
maintain their sample. For example, Alamprese et al. (2003) were able to maintain 70% 
of a sample of 920 learners over a nine-month period. However, adult literacy students 
are often not available for testing because of sporadic attendance and unannounced 
dropout. This leads to many subjects who have been given a pretest but who are not 
available for a posttest. Often, these dropouts cannot be contacted, and this leads to high 
levels of attrition from the study. The time spent in instruction between pretest and 
posttest is often different among individuals in a population of subjects as well, since 
attendance patterns are inconsistent. High attrition rates and inconsistencies in the amount 
of treatment can make findings useless or at least less useful. 

This condition is the result of factors in the lives of program participants that lead 
to dropout and erratic attendance. Some programs address this condition by employing an 
open enrollment policy that allows participants to enter and leave services at any time. 

An adult education class, therefore, might have the same number of students at the end of 
a term as at the beginning, but they might be a widely or even completely different group 
of people. Even when programs employ managed enrollment, students are likely to drop 
out and to have different hours of engagement. 

The problems caused by a small attrition rate or small differences in the intensity 
or duration of treatment can be addressed by employing statistical controls, after 
checking for differences between those who drop out of the study and those who do not. 
However, high rates of attrition are common, even in well-designed, randomized 
experiments in adult literacy research. For instance, a 1994 study of the adult literacy 
classes in a welfare-to-work program, which involved a sample of 1,1 15 clients who were 
randomized and followed up with appropriate data collection, experienced an attrition 
rate of 35% (Martinson & Friedlander, 1994, cited in Brooks, Torgerson, and Porthouse, 
2003). As Brooks et al. (2003) point out, attrition rates of this magnitude can lead to a 
bias in results if the attrition is not random. 

Some of the studies reviewed by Beder (1999) suffered from both high attrition 
rates and treatment issues. For example, the National Evaluation of Adult Education 
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Programs (Young, Lleischman, Fitzgerald, & Morgan, 1995) study began with pretests 
for 8,581 subjects but ended with only 1,919 posttested subjects, only a little above 22% 
retention. Due to a number of validity issues, including the fact that a group of ESOL 
students had more than 50 hours of instruction prior to taking a pretest, the final analysis 
of learning gains was conducted with a sample of only 614 subjects, roughly 3%of the 
total potential test sample (Young, Fleischman, Fitzgerald, & Morgan, 1994). 

The Tennessee Longitudinal Study (Bingman et al., 1999) set out to understand 
how participation in literacy programs changes adults’ quality of life and focused on 
adults with low levels of literacy skill. Data collected in the first year were to serve as a 
baseline for comparisons with subsequent years. During the first year, while researchers 
planned to interview 240 learners, only 133 qualified for the study. Longitudinal 
comparisons were to be made in the second year. However, attrition resulted in a sample 
of only 70 in the second year. In addition, the composition of the sample changed 
substantially with respect to geography, race, and employment status. Second-year data 
were to be collected one year after the initial data collection; however, in reality, follow- 
up data were collected between 12 and 20 months after the baseline data (Beder, 1999, p. 
50). Because of such factors, as well as the withdrawal of funding needed to complete the 
study, the analysis that researchers were able to conduct and the inferences that could be 
drawn from the data were quite limited. 

One option for keeping attrition rates to a minimum is to invest substantial 
resources for keeping participants connected to a study. Reder (2006) reports that the 
Portland longitudinal study maintained 93% sample retention with approximately 1,000 
high school dropouts over six years with significant effort and investment. When subjects 
were recruited, they were told that they were a source of information and expertise, rather 
than a population to be studied. This may have built a sense of obligation and 
commitment to the study. Subjects were also paid $35 for the first wave of data, and that 
payment increased to $50 for the fifth wave of data collection. This payment acted as an 
incentive for participation, but it also served as an acknowledgement of the important role 
the subjects were playing in the study. As they met with study participants, members of 
the research team worked to establish and maintain rapport, an element that has been 
important to their success in maintaining the sample (S. Reder, personal communication, 
June 8, 2006). 

Researchers made multiple efforts to stay in touch with participants between data 
collection periods. This helped to maintain rapport and provided useful information, 
including specific contacts, for locating participants for the next wave of data collection. 
Staff called participants every three months between data collection waves and sent 
periodic postcards to “say hello” and stay informed of address changes. They also sent 
out a newsletter once a year. In addition, the study used a “variable incentive” (lottery) to 
encourage participants to stay in touch with researchers between waves of data collection. 
When there were two years between interviews, the research team held a raffle. In the 
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newsletter, the team announced that individuals who provided up-to-date contact 
information by a certain date would be entered into a raffle in which the team gave away 
$100 each to five people (C. Strawn, e-mail communication, June 20, 2006). 

Substantial efforts were also made to keep in contact with subjects who were not 
immediately available for data collection. Participants voluntarily gave researchers 
contact information to use in the event that a participant could not be located. In some 
instances, staff was unable to locate contacts and so would make use of Web-based 
services to locate the missing participant. Participants gave permission for such a process, 
and it was made clear that if at any time participants were uncomfortable with such a 
search, they could pull out of the study. In cases where individuals had moved too far 
away for the team to collect data in person, the research team would undertake a phone 
interview and hire a local researcher to do the hands-on portion of data collection. 

The Portland longitudinal study benefited from the hard work and significant 
investment in retaining the sample built into the study design. In addition, graduate 
students were well trained and highly committed to doing a good job. Much of the 
success in maintaining the sample is attributable to their efforts (S. Reder, personal 
communication, June 8, 2006). 

Since the resources to maintain a large sample over time may not be available, 
some studies employ a limited time period for the data collection. Several studies have 
limited their collection time to four months but have been able to identify effects. 
Comings, Parrella, and Soricone (1999) explored persistence among 150 students in pre- 
GED classes in New England. Students were interviewed in order to understand the 
factors that promote and hinder their persistence. The first round of interviews took place 
between October and December. Four months after the first interview, researchers 
attempted to contact all study participants and successfully interviewed 79% (119) of the 
original sample. Even with this short time period, the study was able to identify several 
factors that had statistically significant positive relationships to persistence: being age 30 
and over; having teenage or adult children; having been schooled outside the U.S.; and 
stating a clearly defined goal at the beginning of the study. 

In their study of mid-level readers, Strucker, Curtis, and Adams (in press) tested 
subjects at the beginning and end of a semester (after 30 hours of instruction) in order to 
observe changes in students’ reading levels, based on four variations in approach to 
reading instruction. Subjects were also tested approximately one year later to see if gains 
were maintained. Of the 274 students pretested, 205 persisted in their classes long enough 
to be posttested at the end of the semester. Subjects are also undergoing follow-up testing 
one year later (J. Strucker, personal communication, December 6, 2006). 
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In the What Works Study for Adult ESL Literacy Students, researchers assessed 
subjects at the start of instruction and at three- and nine-month intervals, regardless of 
whether or not individuals continued to attend ESL classes. Of the 495 students pretested, 
356 (72%) took the second assessment and 263 (53%) took the third set of assessments 
(Condelli, 2002). Using a growth modeling approach, researchers found that three 
instructional strategies were related to growth in student literacy and language learning: 
“connection to the outside world, use of the student’s native language for clarification in 
instruction, and varied practice and interaction” (p. 27). 

Most studies address the problem of inconsistent time-on-task by collecting 
contact hours. Ranges can be quite wide, but statistical processes can account for these 
differences. In a study of adult learner persistence in library literacy programs, Comings, 
Cuban, Bos, and Porter (2003) report that learners averaged 57 total hours of 
participation over an 18-month period. Students ranged substantially in their 
participation, with 21% having over 80 hours of activity and 24% having 10 hours or less 
of participation. Monthly participation ranged from 3 hours or less to 13 hours or more. 

In the final report of the study (Porter, Cuban, & Comings, 2005), researchers report that 
participation of a sample of the students ranged from 2 hours to 744 hours in tutoring, 
classes, or computer lab activities over a 14-month period. Ultimately, no significant 
relationship was observed between time spent in attendance at a literacy program and 
literacy achievement. Study authors concluded that most students are not persisting 
enough “to make meaningful gains on standardized tests” (Porter et al., 2005, p. 25). 

A study by Comings, Sum, and Uvin (2000) explored the relationship between 
learning gains to hours of instruction over the course of a program year in Massachusetts. 
The range of hours of instruction varied from 10 to over 200 hours, with a median of 74 
hours. In its descriptive data, the study revealed that “as students receive more hours of 
instruction, they are more likely to achieve learning gains, and the average learning gains 
increase with more hours of instruction” (p. 61). Statistical analyses demonstrated that 
hours of instruction are positively related to the probability of obtaining a high school 
credential as well. 

In their study of reading instruction for low-level adult learners over a one 
program-year period, Alamprese et al. (2003), used multiple measures of attendance, 
including average hours that students attended reading classes, the number of weeks 
students attended class, and the percent of the total hours of class that students attended. 
Students averaged 124 hours of instruction over a period of 15 to 37 weeks. On average, 
the authors report that students attended 63% of the class hours available to them. While 
the study did not find a clear connection between hours of instruction and reading gains, 
results did show differences between learners who had attended class for more than seven 
weeks compared to those who had attended up to seven weeks. Patterns differed for non- 
native and native-born students. 
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The patterns of participation of students in adult literacy programs make some 
types of research difficult. However, several researchers have found ways to lessen the 
sample attrition and variations in time-on-task. Study design must allocate sufficient 
resources to sample retention and time-on-task must be carefully recorded so that it can 
be taken into account during analysis. 



Inaccurate Data 

One way to lower the cost of data collection is to draw from sources of data collected by 
programs for administrative, accountability, or planning purposes. However, Beder 
(1999) found that, in adult literacy programs, this data is often incomplete. He found that 
program data had many problems including, “inaccurate learner records, failure to pre- 
and posttest at specified intervals, administration of inappropriate levels of tests, failure 
to test, high attrition of subjects between pre and post data collection, programs’ 
withdrawal from the study before program data collection was complete, and failure to 
forward data to researchers in a timely fashion” (p. 1 16). 

Beder suggests that inaccurate data is caused by the lack of program staff trained 
in testing and data collection. In addition, he suggests that some program staff fear that 
testing and data collection may alienate learners who have a history of failure with tests 
when they were in the K-12 system or have reasons to keep some personal data secret. In 
addition, the high dropout rate and inconsistent attendance make keeping complete 
records difficult. 

Researchers sometimes overcome this problem by collecting their own data, but 
this adds to the expense of a study. In addition, since any data collection in the adult 
literacy environment is difficult, researchers should perform an extensive review of the 
data they collect to ensure accuracy of data collection and results. One approach to 
address this problem could be to verify the program data by collecting a sample of the 
same data that the program is collecting to check for accuracy. However, no example of 
this approach was found in the published literature. 

Even when the study collects its own data, researchers should check for accurate 
data collection and reporting. The Adult Reading Components Study (ARCS) 
administered reading components tests to almost 1,000 adults in ESOL and ABE 
programs. In the ARCS study, researchers reviewed audio tapes of subject interviews, as 
well as test administration and results, including teachers’ assessment of students’ grade 
level equivalents. While time consuming, this process allowed the researchers to assess 
both the quality of the interviewing staff and the validity of results obtained in testing (R. 
Davidson, personal communication, May 31, 2006; Strucker, Davidson, & Hilferty, 2000) 
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Another approach to the problem of inaccurate data is to provide training to 
program staff and support to the development of efficient data entry systems to improve 
program data. For example, the Literacy in Libraries across America Initiative (Comings 
et al., 2003; Porter et al., 2005) faced the challenge of combining data across multiple 
programs, when data were collected differently and at different times by each program. 
The initiative aimed at increasing student persistence in 15 library literacy programs 
around the country. Researchers found that the operational meaning of terms including 
“active,” “program entry,” and “program exit” differed across programs, depending on 
each program’s interpretation of the term. 

In some programs, students were considered “active” if they attended activities in 
a given period. In other programs, however, students were considered “active” if they had 
not communicated that they had dropped out or if they maintained communication with 
the program without actually participating in activities. In order to develop a common 
measure across programs, researchers developed a definition that would work across 
multiple programs. A student was defined as active “in any given month if she or he had 
at least one hour of participation recorded in the database for that month” (Porter et al., 
2005, p. 68). Activities could include any activity for which attendance was recorded and 
generally included some combination of tutoring, class, or group sessions, as well as 
computer lab activities. 

Similar challenges arose for defining program entry. Some programs did not 
record a start date for students, while others recorded the date of first contact or the date 
of the first class or tutoring session. Researchers needed a common definition for first 
month of entry, which they decided on as the “first month in the database in which a 
student was ‘active’ after a period of three or more months of not being active” (Porter et 
al., 2005, pp. 68-69). “Program exit” was defined by a pattern of student participation, 
rather than a particular date. As such, the month of exit was defined as “the first month 
following program entry before at least three months of no activity” (Porter et al., 2005, 
p. 69). Once these definitions were set, the researchers trained the program staff to 
accurately input data, but in some cases this required new hardware and software 
supplied by the study. In this case, the added expense of training and support was 
essential for accurate data collection. 

Despite efforts to create these common definitions, the data had several 
limitations. For example, in some instances, the month of program entry may not actually 
have been a student’s first month in the program, since some students may have been 
active in the program several months earlier. With respect to program exit, the study’s 
definition may not have captured exits without a return, and instead captured the end of 
the current spell of participation. Consequently, the study report notes that “exit rates 
may slightly underestimate the actual percentage of students who actually left a program” 
(Porter et al., 2005, p. 69). 



23 




NCSALL Occasional Paper 



January 2007 



Since Beder’s review of previous studies in 1999, programs have improved their 
data collection as part of the accountability demands of WIA. However, this data often 
needs cleaning and preparation before it is useful for analysis, according to one of the 
reviewers of an earlier draft of this paper. Some programs probably now have data 
collection systems sufficient for research, though some may not. Choosing programs that 
already have good data collection systems, being clear as to definitions of the data being 
collected, providing incentives, and then adding training and assistance to that system 
may be sufficient for most research studies. This approach would be less expensive than 
budgeting for all data collection for a study. 



Lack of Valid and Reliable Tests 

Beder (1999) identifies the lack of valid and reliable tests as a challenge to research in the 
adult literacy field. Research only works well when the expected outcomes of education 
are clearly defined. Without a clear definition of the outcomes and good instruments to 
measure those outcomes, the validity and reliability of a study’s findings may be suspect. 

Most of the instruments employed in the field were developed to guide instruction 
or measure outcomes for accountability systems, not to measure outcomes for research 
purposes. The existing tools may not measure the type of impact that is taking place or 
may not measure the small increments of improvement that occur over the limited time of 
treatment. The use of the same tests for research that are used for other program purposes 
may overestimate improvement since subjects become more familiar with the tests. This 
might not be a problem with experimental designs in which treatment and control groups 
are both developing familiarity with the tests. However, in some studies, such as 
longitudinal panel studies in which some subjects are in programs and some are not, test 
familiarity causes problems. 

Researchers who conducted the What Works Study for Adult ESL Literacy 
Students concluded that the “assessments available for these students for instruction, 
research and accountability purposes range from non-existent to inadequate” (Condelli, 
2002, p. 33). As part of their effort to capture literacy gains among low-level learners, 
researchers used a combination of standardized tests, including the Adult Language 
Assessment Scales (ALAS), the Comprehensive Adult Student Assessment System 
(CASAS) Lunctional Writing Assessment, and alternative assessments. The alternative 
assessments the team developed included a reading demonstration assessment, used to 
“measure students’ ability to read and understand authentic materials they encounter in 
everyday life” (p. 33). Materials included food labels, flyers, simple stories, and 
newspaper articles. Researchers concluded that such an approach is promising and might 
serve as the basis for future development of more suitable assessments to measure 
changes in the reading development of low-literate ESOL students. 
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As part of their study of persistence in library literacy programs in the U.S., Porter 
et al. (2005) included an assessment of literacy skill development for program 
participants over a 14-month period. Since no single test was available to measure the 
multiple aspects of literacy development — decoding, vocabulary, fluency, and reading 
comprehension — the study used a series of standardized assessments, which researchers 
felt were suitable for the study, and relied on different procedures for different students, 
thereby making the standardized assessments “more appropriate for each individual, and 
thus more meaningful and valid” (Porter et al., 2005, p. 58). These tests included the Test 
of Word Reading Efficiency (TOWRE), The Peabody Picture Vocabulary Test (PPVT), 
the reading comprehension portion of the Adult Basic Learning Examination (ABLE), 
and the Basic English Skills Test (BEST) administered only to English language learners 
in the study. 

Similarly, in their study of reading instruction for low-level learners in adult 
literacy programs, Alamprese et al. (2003) employed multiple standardized reading tests 
to measure the development of reading skills among study participants. The six-part 
battery included the Woodcock-Johnson revised word attack and letter-word 
identification, WRAT3 word reading and word spelling segments, and the Nelson word 
meaning and reading comprehension segments. 

In order to examine the literacy growth of adults over time, the Portland 
longitudinal study needed an assessment instrument to measure changes in basic skills 
over several years. The criteria for selecting such an instrument were that it be “(1) 
psychometrically sound; (2) available in multiple forms and appropriate for measuring 
skills growth over time in the target population; and (3) appropriate for and easy to 
administer in an in-home context” (NCSALL, 1998, p. 33). In addition, the instrument 
needed to be one not regularly used by adult literacy programs to avoid some adults 
having exposure to the test through participation in adult literacy programs, and the 
instrument needed to assess literacy proficiencies commensurate with those used in other 
large-scale assessment studies, such as the National Adult Literacy Survey. 

Given these criteria, the research team chose the Test of Applied Literacy Skills 
(TALS) as the standardized basic skills test. To reduce the amount of time needed for 
administration of the assessment, researchers decided to use only the TALS document 
literacy scale, since this scale was highly correlated with the prose and quantitative scales 
of the test. Either Form A or B of the TALS was randomly assigned and administered in 
the five waves of data collection. Two other instruments were included in the assessment 
design of the Portland longitudinal study. The WordSum oral vocabulary assessment was 
used in the second wave of data collection, and a writing sample and the Test of Word 
Reading Efficiency (TOWRE) test of fluency and decoding skills were each added for the 
fifth wave of data collection. 
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As part of ongoing national Birth Cohort Studies, the National Research and 
Development Centre (NDRC) in the U.K. added literacy assessments to recent waves of 
data collection. These assessments, designed or adapted for the study, include tests of 
reading and numeracy based on the national adult literacy curriculum, writing samples, 
and scales for people to examine learning differences, or learning disabilities (S. Reder, 
personal communication, June 8, 2006). 

The examples described here reflect the need for the development of an approach 
to assessment that comprises a multiplicity of measures. For instance, literacy 
assessments are required that can assess both the components of reading (e.g., word 
recognition, decoding, vocabulary, and comprehension) and accomplishment of tasks 
with reading skills. One option is to identify a set of existing instruments suitable for 
research and support the development of additional instruments to provide a wider range 
of tools. Some of these tools would measure changes in skills and some would measure 
changes in ability to perform tasks. A commonly accepted set of assessment tools that 
were only used for research could overcome this challenge and make meta-analysis easier 
as well. 



Multiple Populations 

The adult literacy population is often treated as a homogeneous group, but, in fact, it is 
diverse in many ways that might affect research findings. A treatment that works for one 
group may not work for another, but a study might show an average effect across a 
population with wide diversity. Even when research is restricted to a specific population, 
important differences within the population may not be identified. For example, 
beginning ESOL students who are literate in their native language may respond 
differently from those who are not literate in their native language (Condelli, 2002). 
Unfortunately, individual ESOL classes may have students in both categories who are not 
identified as to their literacy skill background. Research that seeks a random sample of 
the entire population may not be relevant to one of the subpopulations. Studies could 
enlarge their sample size to include enough subjects from each subpopulation or could 
limit its sample to specific populations. 

Research in K-12 schools must address the same challenge, but that research has 
well-defined subpopulations. Adult literacy researchers might also define subpopulations 
based on specific skill sets, and then suggest that researchers report findings based on 
these subpopulations. Some researchers have suggested such subpopulations. 

Sabatini (1999) suggests four skill groups: 

1. Truly nonliterate students who cannot read at all. They are rare in the U.S. 
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2. Low literate students who do not read well. They probably dropped out of 
school or did not read and write enough while in school to develop strong 
literacy skills. They are probably the largest group of adults who could benefit 
from adult literacy services. 

3. Reading disabled students who have trouble reading. They failed to develop 
strong literacy skills because of learning disabilities or cognitive disabilities. 
Though this group is probably between 5% and 10% of the adult population, 
they may make up a larger percentage of the group of adults who could 
benefit from adult literacy services. 

4. Second language students who cannot read in English. They may be illiterate in 
their own language; they may be literate in a non- alphabetic language, or they 
may be literate in an alphabetic language but not speak English. This group 
may be less than 5% of the U.S. population, but they are at least 50% of the 
students in WIA-funded programs. (Strucker and Davidson (2003) found that 
among Spanish speakers in a sample drawn from WIA-funded programs [31% 
of the sample], 80% had adequate or better than basic Spanish literacy skills.) 

Strucker and Davidson (2003) identify ten clusters of adult students based on 
extensive testing of their English component reading skills. These clusters fall into 
three groups: 



GED and Pre-GED 

1. Strong GED 

2. Pre-GED with vocabulary and background information needs 

3. Pre-GED with vocabulary, spelling, and fluency needs 

Intermediate 

4. High intermediates with difficulties in print skills and fluency 

5. Intermediates with stronger print than meaning skills 

6. Intermediates with low fluency 

7. Low intermediates 

8. Low intermediates who have weak English language skills 
Low Level and Beginning 

9. Beginners 

10. Reading and fluency impaired 
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Comings (2003) identifies six populations of adult literacy program participants: 

1. ESOL students who are literate in their native language and test at or below 
SPL4 3 on the BEST. 4 

2. ESOL students who are literate in their native language and test at or 
above SPL5. 

3. GED 5 students whose literacy skills are at or above the 8 th grade level as 
measured by a test such as the TABE 6 or CASAS. 7 

4. ABE students whose literacy skills are between the 6 th and 8 th grade level. 

5. ABE students with literacy skills below the 6 th grade level. 

6. ESOL students who are not literate in their native language. 

Sometimes classes made up of a single skill group can be formed for small-scale 
research studies, but in most programs classes are mixed. In those cases, research 
analysis would have to separate out the data into subgroups, which might require a 
larger initial sample size. 



Lack of Program Resources 

Greenberg (2007) identifies the lack of program resources as a challenge to research in 
the adult literacy field. This is a challenge to research because it undermines successful 
implementation of interventions. This issue arises in K-12 research as well, but the low 
level of funding provided to adult literacy programs exacerbates this situation. According 
to data from the Office of Vocational and Adult Education, the average annual 
expenditure per student in fiscal year 2002 was approximately $800. The range, among 
the 50 states, was between $208 and $2,683; 16 states were below $500, and only 9 states 
were above $1,000 (U.S. Department of Education, 2005). 

Because of low funding levels, most adult literacy programs lack the resources 
needed to participate in a research study. For example, they often do not have enough 
space to provide room for interviews or staff trained to administer tests. Greenberg 
(2007) offers an example of a teacher whose class was considered for participation in a 
study. Due to limited space, the teacher taught her classes in the same room that 
contained her office. Consequently, she was regularly interrupted with phone calls and 
other intrusions. The teacher kindly offered to allow the researcher to carry out the 



3 The BEST has 10 Student Performance Levels. 

4 The Basic English Skills Test is published by the Center for Applied Linguistics in Washington, D.C. 

5 The General Educational Development test provides certification of skills and knowledge that 
approximates those of a high school graduate. 

6 The Test of Adult Basic Education is published by McGraw-Hill in New York, NY. 

7 The Comprehensive Adult Student Assessment System is published by CASAS in San Diego, CA. 
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desired intervention with a class within this room, but the researcher had to decline the 
offer, deeming the space inappropriate for research. 

Another resource-related challenge to conducting research is that turnover among 
staff is high in programs that cannot provide stable employment. In addition, since 
funding is sometimes unstable, programs can disappear in the middle of a research 
project or decide that they can no longer afford to participate. A study by Purcell-Gates, 
Degener, Jacobson and Soler (2000) reflects this challenge. The study, which investigated 
the relationships between two dimensions of adult literacy instruction and change in the 
literacy practices of adult literacy students, involved data collection from 83 adult literacy 
classes in 22 states. Management of these multiple sites proved challenging to the 
research team. In some cases, programs closed before data could be collected. In one 
instance, a program “closed and reopened under new directorships three different times 
before a data collector could ever gain access to the site” (p. 24). 

One solution is to provide additional resources to programs that participate in 
research projects. This, of course, causes an increase in research costs, especially since 
control or comparison groups might need access to a similar level of new resources. 
Significant differences in access to resources might result in important differences in the 
nature of treatment and control groups, which would, therefore, make it impossible to 
rule out rival explanations for the relationships observed in a study (Light, Singer, & 
Willett, 1990). Research reports would have to identify the greater resources as an 
element of the intervention and explain that, without these additional resources, programs 
may not be successful in following advice based on evidence from the study. 



Sampling 

Because of a lack of resources, most adult literacy studies employ a convenience sample, 
but a probability sample provides a much stronger basis for analysis. Even when 
sufficient funding is available, defining a sample may be difficult. The first step in 
defining a sample usually involves obtaining a list of programs and within programs a list 
of teachers or students. Unfortunately, these lists sometimes do not exist or may be 
inaccurate or misleading. In some cases this situation is caused by inadequate data 
systems or untrained personnel. Even when the data systems are sufficient, they may be 
difficult to interpret. For example, programs, teachers, and students sometimes are 
defined by the funding that supports them, and this can lead to double (or more) counting 
of the same program or person, because they are funded by more than one source, or the 
mislabeling of them, since they are funded by a source that supports services different 
from those being offered. 

Researchers use an array of tactics to create feasible and meaningful research 
samples. In a recent study, Waterman (2006) randomly selected adult ESOL classes to 
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form treatment and control groups in her study of the effectiveness of an intervention to 
promote parent involvement and improve ESOL skills. To arrive at these groups, the 
researcher used multiple selection criteria. Within one school district, she sampled 
elementary schools with at least 50% of the students having immigrant parents and more 
than 85% of students with low socioeconomic status. In addition, she limited selection to 
schools whose principals expressed interest in supporting and receiving the intervention 
and agreed to provide a staff member to act as a liaison among the school, the ESOL 
teacher, the parents attending class, and the researcher. 

These steps ensured equal support for adult ESOL classes in both treatment and 
control groups. With a group of schools that met these criteria. Waterman then randomly 
selected a single ESOL class from each of four different schools to be the treatment 
group and performed a similar process for selecting the control group. Classes in the 
treatment group received the intervention while those in the control group did not. 
Waterman limited her final analysis to women of Mexican origin who had at least one 
child in the participating school, which resulted in a sample size of 66. The study 
findings, therefore, might have been different with ESOL students who were not Mexican 
mothers who had children in schools with less accommodating principals. 

To provide for the equivalence of the treatment and control groups, Waterman 
made an effort to ensure the average equivalency between the groups of mothers included 
in the study, as well as the adult ESOL instructors involved in the study. This equivalence 
was determined using data on students’ socioeconomic status, date of arrival in the U.S., 
level of formal education, hours of attendance in the ESOL class under study, and prior 
experience in ESOL instruction. For teachers, she looked at formal training related to 
adult education and ESOL teaching, prior experience teaching adult ESOL, and prior 
experience with parent involvement and immigrant parents. 

Sampling procedures used in the Adult Reading Components Study (ARCS) 
provide another example of how the challenges of selecting a sample of study 
participants from adult literacy classes can be managed (Strucker et al., 2000). The study 
was the first large-scale attempt to describe the reading of students enrolled in ABE, 

ASE, and ESOL classes. It used a battery of individually administered reading and 
language tests. In order to select program sites, the research team first focused on 
obtaining the agreement of eight state directors to support the research process. The team 
next obtained lists of programs offering ABE, ESOL, or ASE services within each of 
these states. From that list, the researchers focused on programs serving at least 100 
students, for reasons of economy. A sampling statistician used information on the 
proportion of ABE and ASE to ESOL classes and the native languages spoken by the 
ESOL students to select from state lists programs that accurately represented the mix of 
students within each state. The study director personally contacted all program directors 
to explain the study, and, ultimately, 80% of those programs contacted agreed to 
participate. These conversations provided important insights into the nature of the 
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programs and the actual number of students at each program, which often differed from 
state-level figures. Without this firsthand information, the sample might have been 
different than originally projected. 

Next, researchers faced the task of selecting individual students for inclusion in 
the study. Originally, this process required program directors to provide a list of all 
students in each school (with the exception of non-Spanish- speaking ESOL students, 
who were not included in the study). The team’s sampling statistician would then 
randomly select 30-50 names for each center, depending on the center’s size. 

Researchers then set out to administer tests to those students selected. The realities of 
conducting research in adult programs quickly confronted researchers. Researchers would 
arrive at a learning center ready to test those selected students, only to find that most of 
these individuals were absent that day, had changed classes, or had dropped out in the 
two weeks between selection and testing. As a result, far fewer students than planned 
were tested, and resources were wasted on testers who had no students to test. 

To overcome this problem, the research team changed its approach to sampling by 
selecting participants from among students actually present on the day of testing through 
a lottery. The number chosen by raffle varied to help ensure that subjects would be 
sampled in proportion to their numbers. So, for example, from a class of 20 students, the 
team would select two lottery “winners,” while from a class of 10 students, only one 
“winner” would be chosen. Teachers were instructed to announce the testing to their 
classes one or two days prior to the testing only by indicating that some researchers 
would be coming to talk about some tests they were giving, that participation was 
voluntary, and that students would be paid for their involvement. This description was 
neutral in tone so that students could decide for themselves whether or not to participate. 

Based on this experience, the research team offered five key recommendations for 
future studies in adult literacy. First, the students should be paid for their participation. 
Second, students should be tested during their scheduled class hours, using some form of 
on-the-spot lottery. Third, teachers or administrators should neither select nor influence 
the selection of students or classes that are sampled. Fourth, researchers should always 
meet personally with teachers and administrators to explain the study in detail before 
beginning data collection. Finally, “principal investigators should make themselves 
personally available to teachers, administrators and students” (Strucker et al., 2000, p. 29). 

As data collection improves under WIA, lists of programs and lists of teachers and 
students within programs should become more available and more accurate. Even so, given 
the nature of adult participation in programs, final sample selection may have to take place 
at the program level, and so studies should plan for this added time and expense. 
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Shortcomings of Existing Data Sets 

One way to avoid the time and cost of collecting data is to employ data sets already built 
by other studies. While these data sets provide rich sources of data on which to conduct 
research (and are, in fact, cited as such in the section on advantages), they are often 
lacking in key variables of interest to researchers in adult education. 

For example, in her study of the labor market performance of Spanish-speaking 
dropouts who obtained a Spanish GED, Soricone (2003) used a unique data set that 
combined GED testing data with Unemployment Insurance administrative records. While 
these data sets permitted a longitudinal analysis of earnings related to skill levels, neither 
set contained variables, such as immigrant status, age at immigration, location of 
previous schooling, or measures of English proficiency, which would have been useful in 
interpreting study findings, as well as in answering additional research questions. Such 
measures have been identified by other researchers (Wrigley, Richer, Martinson, Kubo, 

& Strawn, 2003) as important to developing our understanding of the activities and 
outcomes of limited English-proficient adults in the U.S. 

An additional challenge is that administrative records of the Unemployment 
Insurance system have an inherent limitation in what they can tell us about earnings. 
Periods of “zero earnings” could in fact reflect unemployment, employment in an 
underground economy, self-employment, incarceration, departure from a state or the 
U.S., and even death. Thus, the true nature of immigrant earnings may be misrepresented 
with the use of such data. At this point, better sources of earnings data are not available. 

The National Household Education Survey (NHES) provides valuable insight into 
adults’ participation in adult education as well as earnings. One drawback to this data set, 
however, is that the information is self-reported and may, therefore, overestimate actual 
earnings (Hollenbeck, 1996; Uvin, 2001). Furthermore, the NHES does not include adults 
who live in households without telephones, institutionalized adults, and adults in the 
armed forces. Consequently, certain ethnic, racial, or linguistic groups may be 
underrepresented. Another problem with these data is that they do not include important 
information on workers, such as age at immigration for immigrants, and firm size, which 
would allow researchers to more appropriately attribute some of the variation in workers’ 
earnings (Uvin, 2003). With respect to the information on participation in adult education 
that NHES can provide, these data lack measures of, for example, the supply of available 
ESOL classes, as well as indicators of individuals’ status on waitlists (Uvin, 2001). 

Other national data sets have limitations that impact the nature of research that 
can be undertaken using them. For example, the sample from the High School and 
Beyond (HS&B) data set used by Murnane et al. (1999) includes individuals who began 
the study in 10 th grade. Consequently, while the data set provides a rich source of 
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longitudinal data, it excludes individuals who may have dropped out prior to 9 th grade, a 
population that is likely to be represented in adult literacy programs. 

Nothing can be done about missing variables in data sets that already exist. Adult 
literacy researchers, though, could identify the data elements they need and promote their 
inclusion in future studies. 



Lack of Research Resources 

Just as the adult literacy field lacks resources to support and improve programs, it lacks 
resources to fund research. Direct funding for research in adult literacy probably amounts 
to less than $10 million a year. Research funding available from the Institute of 
Education Sciences and the National Institute for Child Health and Human Development 
is mostly focused on K-12 children. Most foundations that support education research are 
also focused on K-12 children. Some government agencies that address such issues as 
workforce development or welfare include adult literacy in their research funding but 
usually as a component of a larger program effort. For example, the National Evaluation 
of Welfare-to-Work Strategies was designed to offer insight into the effectiveness of 
different types of welfare-to-work programs. While the study provided information on the 
economic and other outcomes of program participants, it also offered suggestions for 
improving program design for educational programs that work with welfare recipients 
(Hamilton, 2002). 

This lack of funding discourages some academics from dedicating all or part of 
their career to adult literacy. This makes it difficult for potential graduate students to find 
a mentor who could help them develop an academic career. These constraints have 
limited the establishment of peer-reviewed journals and other publication opportunities. 
This cycle of constraints has produced a limited scholarly community. 

That limited community has made attempts to develop itself. The Adult Education 
and Adult Literacy Special Interest Group (SIG) within the American Educational 
Research Association has grown from less than 30 to almost 200 members over the last 
six years. The SIG offers a venue in which a scholarly community can be built. Adult 
literacy SIGs have also been established in the International Reading Association and the 
National Reading Conference. 

The National Institute for Literacy (NIFL) is developing a comprehensive agenda 
for research and development in adult literacy, and though it cannot fund a large research 
effort, its leadership could expand resources for the field. Collaboration among NIFL, 
IES, NICHD, and the AERA SIG, therefore, may be one way to work toward expanded 
resources for research. This could lead to a larger scholarly community, which could then 
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support peer-reviewed journals. With a stronger research community and a clear agenda, 
funding might increase. 




OPTIONS FOR SUPPORTING RIGOROUS RESEARCH 



Individual researchers could address these challenges on their own. However, the dialogue 
among the advisory group identified options for the adult literacy field to work together to 
support rigorous research. These options include: (1) a more ambitious effort to describe 
and develop approaches to overcoming the challenges to scientific research in this field, 
(2) larger studies, and (3) promotion of adult literacy programs as allies in research. 



A More Ambitious Effort 

The group of advisors suggested that this monograph could only begin a dialogue, not 
solve the problems of challenges to rigorous research in this field. They suggested that a 
more ambitious effort could produce a detailed approach to implementation of rigorous 
research in the adult literacy field. If the leadership of the field — funding agency staff, 
researchers, and practitioners — undertook a more ambitious effort, they might lay the 
foundation for a system of research that supports improvement of program services. 

This collaboration might begin its efforts with the description of theory and the 
models that could derive from it. This effort could build a consensus about models for 
services. Once those models were established, the research community could pursue a 
research agenda that would test and improve these models as well as explore new models 
that might prove to be more effective. 

An ambitious effort might both identify more challenges and provide resources 
for scholars to expand the sections on each challenge examined in this monograph. This 
would require more resources than were available for this monograph, but that 
investment might set the stage for more efficient use of the limited research funds that 
are available. 



Larger Studies 

Since the challenges to scientific research that the adult literacy field faces require 
resources to overcome, researchers might consider pooling their resources in studies that 
answer several questions with the same data collection procedure. This would allow more 
resources to be put into design and implementation of data collection, which would then 
allow studies to address the many challenges to rigorous research in this field. One large 
study that answers five questions might be more likely to succeed than five smaller 
individual studies. 

The economies of scale that larger multi-question projects would provide might 
also increase the efficiency of the use of funds available for research. Several funding 
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agencies might work together to pool resources and derive more research, and research of 
a higher quality, than they could manage on their own. Such collaboration is taking place 
in a program of research that includes six studies focused on adult literacy that are funded 
and managed by NIFL, NICHD, and OVAE. In addition, agencies that fund K-12 
research might include cohorts of adults in their studies. Adding a cohort of adults to a 
well-designed study might be less expensive than a separate adult study. 



Programs as Allies in Research 

Since most research in the field takes place in adult literacy programs, practitioners could 
be allies in lowering the challenges to research. At this time, the research and practice 
communities only engage in discourse occasionally. The advisory group suggested 
several ways in which the research community could deepen its contacts with programs. 

The advisors suggested that most practitioners probably need training to become 
better consumers of research, but the advisors suggested that this training should strive 
to help practitioners see themselves as active partners in the process of connecting 
research and practice. Practitioners should be encouraged to participate in research 
studies. This encouragement could explain why research is important, how it is not 
evaluation of program efforts, and what changes the programs might have to make to be 
supportive of research. This might also lead to more programs willing and able to be 
good research sites. Practitioners should also be encouraged to adapt research findings to 
their specific contexts and feed questions and hypotheses from their classrooms to the 
research community. 
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This monograph is a small step in a larger effort to improve scientific research in the 
adult literacy field and to encourage more research as well. This step should be followed 
by a dialogue among scholars in the field that expands the insights recorded here. If that 
dialogue is sustained, the field should be able, through trial and error and through 
building on previous experience, to address these challenges in ways that are within the 
limits of the resources available. 
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